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From  2008  to  2013,  American  taxpayers  will  spend  over  30  billion  dollars  to  increase  the 
global  maneuverability  of  the  American  Military.  Central  to  this  investment  is  verifying  that 
dollars  invested  by  the  USAF  Air  Mobility  Command  (AMC)  yield  the  effects  they  were  intended 
to  produce.  Upcoming  programs  include  the  KC-X,  Joint  Cargo  Aircraft  (JCA),  and  a  variety  of 
modernization/recapitalization  (modern/recap)  programs  for  the  C-5,  C-130,  C-130J,  KC-10, 
KC-135  and  C-17.  With  an  increasing  amount  of  AMC  operational  test  workload  on  the 
horizon,  standing  up  a  traditional  test  and  evaluation  (T&E)  organization  seems  like  a  viable 
option.  Unfortunately,  large  personnel  reductions  needed  to  support  modern/recap  make 
garnering  manpower  a  challenging  task  to  advocate  with  the  corporate  Air  Force.  However,  the 
authors  met  this  challenge  by  re-building  an  organization  that  represents  a  direct  investment 
into  modern/recap.  They  conducted  a  six-month  study  and  concluded  that  a  “right  sized” 
operational  test  force  increases  potential  for  cost  reduction,  because  it  is  lean  enough  to  create 
savings  for  Air  Force  modern/recap  programs.  In  order  to  achieve  this  vision,  the  authors  used 
an  emerging  doctrine  called  Effects-Based  Mobility  (EBM)  to  connect  the  joint  fight  with  the 
T&E  Enterprise.  The  authors  also  used  Air  Force  Smart  Operations  (AFS021)  to  pinpoint 
shortfalls  in  the  existing  organization  and  quantify  potential  cost  savings.  When  used  together, 
this  analysis  cut  overhead  and  transformed  the  organization  into  a  predictive,  lean  organization 
that  represents  a  direct  investment  into  Air  Force  Modernization  and  Recapitalization 
programs. 

A.  Application 

This  paper  is  intended  to  report  the  results  of  an  effects-based  and  AFS021  approach  toward  “right  sizing”  a 
T&E  organization.  It  is  mostly  geared  toward  DOD  acquisition  professionals,  especially  in  the  T&E  Enterprise. 
Therefore,  the  concepts  and  terms  provided  assume  readers  have  a  basic  understanding  of  T&E  as  it  applies  to  DOD 
acquisition.  On  the  other  hand,  the  approaches  presented  are  also  deeply  rooted  in  industrial  engineering  and 
operation  management  techniques  and  have  a  wide  range  of  technical  and  business  applications.  Concepts  such  as 
Lean,  Six- Sigma,  and  Linear  Programming  were  used  to  quantify  organizational  performance  and  provide  decision¬ 
making  tools  for  transformation.  Therefore,  readers  are  encouraged  to  explore  the  concepts  presented  in  order  to 
apply  to  other  organizations. 

B.  Background:  Mapping  a  Resource-Constrained  T&E  Environment 

Test  and  Evaluation  is  a  risk-reducing  tool  acquisition  managers  utilize  for  weapon  system  development 
programs.  Testing  gauges  progress  made  when  a  system  moves  from  an  idea  to  a  product.3  Adequate  testing 
ensures  weapon  systems  progress  through  DOD  acquisition  milestones  by  verifying  specification  compliance 
(Developmental  Testing)  followed  by  an  evaluation  of  mission  effectiveness  and  suitability  (Operational  Testing). 
The  ultimate  goal  of  T&E  is  to  ensure  a  system  works  as  intended  before  it  is  provided  to  the  customer. 

The  DOD  allocates  a  variety  of  test  resources  to  support  T&E  objectives  for  weapon  system  acquisition.  From 
wind  tunnels  and  environmental  test  facilities  to  large  test  range  complexes,  test  resources  ensure  weapon  systems 
are  measured  in  a  robust  fashion.  Another  key  portion  (and  arguably  the  most  important)  is  the  T&E  workforce, 
which  provides  test  and  operational  expertise  to  design  and  execute  experiments  that  evaluate  system  performance. 
Testers  ensure  systems  meet  required  specifications,  and  are  effective  and  suitable  in  accomplishing  their  intended 
mission. 

C.  Test  &  Evaluation  Challenges 

Often  times  the  success  or  failure  of  an  acquisition  program  depends  on  its  performance  during  testing.  Program 
managers  (PM)  rely  on  T&E  to  measure  system  performance  to  determine  if  a  system  is  ready  to  progress  from 
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development  to  fielding.  PMs  primarily  fund  T&E,  which  often  takes  the  form  of  support  for  resources,  planning, 
execution  and  reporting.  Some  funding  represents  the  overhead  cost  required  for  upkeep  of  assets  mentioned  in  the 
previous  paragraph.  PMs  integrate  this  funding  into  a  five-year  budget  cycle.  In  this  cycle,  competing  programs 
across  the  services  vie  for  funding  in  front  of  Congress. 

With  this  backdrop,  one  can  see  the  challenges  faced  by  PMs  and  how  this  shapes  interactions  with  the  T&E 
force.  Funding  T&E  is  a  costly  endeavor.  Robust  testing  is  often  the  key  to  finding  and  fixing  critical  deficiencies 
before  fielding  a  system.  However,  with  the  discovery  of  deficiencies  comes  the  reality  of  the  “Bad  Press”  effect. 
Bad  press,  in  the  form  of  negative  test  results,  can  result  in  reduced  confidence  in  system  performance.  Reduced 
confidence  forms  a  perception  of  poor  performance,  which  significantly  affects  the  standing  of  competing  programs 
in  front  of  Congress.  Thus,  a  potential  question  considered  by  nearly  every  program  managers  is, 

“Why  would  I  pay  for  resources  that  could  uncover  system  deficiencies  impacting  future 
funding ,  or  even  the  survival  of  my  program  ?” 

This  is  not  to  say  unethical  practices  form  the  basis  of  PM  strategy.  Rather,  it  is  safe  to  say  the  “Law  of  the 
Jungle”  takes  over  when  it  comes  to  weapon  system  survival.  Keeping  programs  “alive”  often  occupies  the  mindset 
of  PMs,  who  deal  with  a  myriad  of  acquisition  challenges.  Life-cycle  savings  can  sometimes  take  a  back  seat  to 
competing  service  priorities  in  a  resource-constrained  environment. 

With  this  frame  of  reference,  forces  are  easily  aligned  to  perceive  T&E  as  an  entity  that  increases  program  costs 
and  generates  bad  press  leading  to  program  delays,  or  even  cancellation.  The  momentum  to  “field  first”  and  rectify 
later  often  results  in  over  budget  situations  as  systems  require  expensive  retrofits. 

With  PMs  engaging  the  purse  strings  of  the  acquisition  environment,  it  is  easy  to  see  the  challenge  the  T&E 
enterprise  faces  toward  building  an  infrastructure  capable  of  effectively  reducing  program  risk.  From  a  historical 
perspective,  a  series  of  acquisition  reforms  in  the  1990s  steadily  strained  T&E  manning.  During  the  “Lightning  Bolt 
Initiatives,”  policy  makers  placed  greater  stock  in  contractor- supported  testing.  As  contractors  bid  on  T&E  for 
major  programs,  Air  Force  leadership  justified  further  reductions  in  the  T&E  Enterprise  but  did  little  to  explore  the 
secondary  effects  of  this  action  on  weapon  system  cost  overruns.  Making  the  situation  even  more  challenging  are 
the  current  PBD  720  budget  cuts,  where  the  T&E  community  is  experiencing  substantial  cuts  to  its  manpower  force. 
With  aging  national  test  facilities,  leaders  are  again  asking  the  question:  How  can  we  improve  the  effectiveness  of 
T&E,  and  what  is  the  right  size  of  our  Enterprise? 

D.  MAJCOM  T&E:  New  Challenges  Presented  to  Air  Mobility  Command 

Air  Mobility  Command  T&E  is  faced  with  similar  challenges  as  the  command  moves  to  invest  in  modem/recap 
of  its  fleet.  Since  the  days  of  Military  Airlift  Command,  AMC  has  relied  on  a  small  MAJCOM  T&E  Enterprise. 
With  a  manpower  force  of  approximately  99  personnel,  AMC  testers  kept  pace  with  OT&E  obligations,  maintained 
oversight  of  DT&E  programs,  and  met  rapid  fielding  requirements. 


Figure  1 .  The  Current  AMC  T&E  Enterprise  is  shown  above  (99  billets  consisting  of  73  permanent  and  26 
temporary  positions).  The  TE  organization  includes  a  headquarters  location,  a  single  squadron  at  McGuire  AFB, 
and  various  detachments  and  operating  locations  positioned  throughout  the  CONUS. 
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A  single  Squadron  conducted  Follow-on  Test  and  Evaluation  (with  Det’s  and  Operating  Locations  shown  in 
Figure  1),  while  AFOTEC  assumed  responsibility  for  Initial  Operational  T&E  (IOT&E).  Given  the  Low  Density, 
High  Demand  nature  of  mobility  assets  and  manpower,  aircraft  were  not  permanently  assigned  to  the  OT&E 
Enterprise  (as  in  ACC  and  AFSOC).  AMC  T&E  relied  primarily  on  temporary  assignment  of  aircraft  and  aircrews 
for  test  execution. 

For  the  most  part,  the  AMC  T&E  Enterprise  met  AMC’s  T&E  requirements  in  the  1990s  and  early  2000s.  The 
C-17  program  completed  IOT&E  with  AFOTEC,  and  a  robust  DT  community  assumed  responsibility  of  block 
upgrades.  Moreover,  Air  Logistics  Centers  assumed  responsibility  of  incremental  upgrades  for  other  mature 
systems,  such  as  C-5,  C-141,  KC-135,  and  C-130.  This  structure  also  met  congressionally  mandated  workload 
brought  about  by  the  tragic  loss  of  a  T-43  aircraft  in  Dubrovnik,  Yugoslavia  and  a  C-141  mid-air  collision  in  Africa. 

Award  of  the  C-5  modernization  program  required  the  addition  of  20  temporary  positions  to  the  AMC  T&E 
Enterprise.  It  appeared  as  though  the  AMC  T&E  Enterprise  was  appropriately  sized  for  future  workload.  By  the 
year  2000,  Headquarters  AF  T&E  policy  makers  shifted  all  IOT&E  workload  toward  AFOTEC  and  transferred 
approximately  40  positions  from  AMC  to  AFOTEC.4 

However,  the  start  of  the  Global  War  on  Terror  brought  significant  change  to  the  AMC  landscape.  AMC 
investments  grew  in  2001,  and  AFOTEC  deferred  C-17  and  C-130J  workload  toward  AMC  MAJCOM  testers.  The 
AMC  T&E  Enterprise  faced  increasing  workload  which  included  rapid  fielding  initiatives  like  Joint  Precision 
Airdrop  Systems  (JPADS)  and  Mobility  Defensive  System  Testing.  At  the  same  time  a  myriad  of  modern/recap 
programs  rocketed  to  the  forefront  of  AMC’s  priorities.  Modernization  of  the  C-5,  C-130,  KC-10,  and  KC-135 
added  significant  workload  to  the  AMC  T&E  Enterprise.  AMC  investments  from  FY08  to  FY13  are  estimated  at 
$30  billion,  as  shown  in  Figure  2. 5  The  substantial  increase  in  investment,  when  overlaid  with  AMC  T&E 
manpower  uncovers  a  potential  challenge  in  the  ability  of  the  T&E  force  to  mitigate  increasing  investment  risks. 
Such  an  environment  poses  several  questions: 


1 .  Is  the  AMC  T&E  Enterprise  properly  sized  to  mitigate  risk  for  this  investment? 

2.  In  a  resource-constrained  environment  what  changes,  if  any,  are  required? 
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Figure  2.  HQ  AMC  modern/recap  investment  ($34B  FY08-13),  overlaid  with  AMC  T&E  manpower  levels.6 


The  authors  set  out  to  answer  these  questions  by  using  Effects-Based  Mobility  (EBM)  and  Air  Force  Smart 
Operations  21  (AFS021).  EBM  is  an  emerging  air  mobility  doctrine  that  focuses  efforts  on  “Critical  Factors”  that 
generate  effects.7  The  authors  used  EBM  to  build  a  connection  between  the  Joint  Fight  and  the  T&E  Enterprise. 
They  also  used  AFS021  to  look  for  adverse  indicators  of  the  changing  landscape,  shown  in  Figure  3.  AFS021  is 
the  Air  Force  Vision  for  achieving  greater  organizational  efficiency.8  It  combines  the  principles  of  Lean,  Six  Sigma, 
Theory  of  Constraints,  and  other  Operations  Management  techniques  traditionally  utilized  in  manufacturing  and 
industrial  engineering  fields. 
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The  authors  started  their  analysis  by  examining  problem  areas  in  major  AMC  programs.  Two  programs  served  as 
a  model  for  this  analysis.  The  first  program  was  the  C-5  Avionics  Modernization  Program  (AMP).  The  second 
program  was  the  C-130  AMP.  The  authors’  investigation  of  both  programs  uncovered  indicators  of  instability  in 
both  of  these  programs,  primarily  in  the  form  of  cost  overruns  and  schedule  delays.  The  figure  below  shows  the 
current  state  of  C-5  AMP  and  projection  of  C-130  AMP.  C-5  AMP  is  currently  17%  over  budget,  while  C-130 
AMP  is  at  128%  percent  over  budget.9 
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[%]  Program  X 
|  Program  Y 


C-130  AMP  Trajectory 
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C-5  AMP 
17%  Over  Budget 
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Schedule/Time 


Figure  3.  Indicators  of  Program  Instability.10 


With  overruns  exceeding  Nunn-McCurdy  Amendment  breaches  and  garnering  significant  Congressional 
oversight,  it  is  easy  to  conclude  that  these  programs  are  experiencing  instability.  However,  determining  the  T&E 
link  to  these  programs  is  a  more  challenging  task.  Analysis  of  C-5  and  C-130  modernization  programs  determined 
that  both  programs  proceeded  to  test  with  relatively  unstable  designs.  The  unstable  designs  resulted  in  schedule 
delays.  With  schedule  pressures,  program  managers  sought  to  “make-up”  time  by  compressing  test  schedules.  With 
a  schedule  scoped  for  a  given  manpower  level,  accelerating  schedules  increased  T&E  manpower  requirements  and 
strained  resources.  The  C-5  AMP  was  fielded  prior  to  acquiring  critical  knowledge  most  often  collected  during 
testing.  The  C-5  also  fielded  with  excessively  large  lists  of  software  defects.  In  the  end,  this  cascading  phenomenon 
resulted  in  expensive  redesigns,  retrofits,  and  delayed  delivery  to  the  warfighter,  shown  in  Figure  3.  The  C-5 
program,  shown  in  solid  red,  experienced  a  17%  cost  increase,  and  1  year  delay  in  development.  The  trajectory  of 
the  C-130  AMP  program,  shown  in  dotted  red,  depicts  a  128%  cost  increase,  and  a  2  year  delay  in  fielding.11  Now 
note  the  blue  bar  at  the  top  of  Figure  3.  It  depicts  the  level  of  involvement  the  AMC  T&E  Enterprise  has  in  mobility 
development  programs.  Notice  how  the  current  AMC  Test  Force  presents  forces  more  toward  the  back  end  of 
development,  where  the  least  amount  of  influence  in  cost  can  be  made.  This  construct  is  primarily  reactive  in 
nature.  At  this  point  the  authors  recognize  that  Air  Force  Regulations  dictate  that  AFOTEC  takes  a  leading  role  in 
OT&E  involvement,  which  is  not  reflected  in  Figure  3.  What  the  authors  focused  on  was  the  link  between 
MAJCOM  OT&E  involvement  and  program  stability.  The  reactive  nature  of  the  AMC  construct  led  the  authors  to 
investigate  the  link  between  T&E  resources  and  program  performance  in  greater  detail. 

The  authors  used  AFS021  to  conduct  a  “Root  Cause”  analysis  of  data  reported  by  the  Secretary  of  Defense’s 
Director  of  Operational  Test  and  Evaluation  (DOT&E)  in  its  annual  report.  The  analysis  mapped  critical 
improvement  areas  identified  by  DOT&E,  and  found  compelling  results  for  mobility  programs.  DOT&E  looked  at 
27  Major  Air  Force  programs  in  2005  and  came  up  with  the  major  improvement  areas  shown  in  Figure  4. 
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As  expected,  hardware,  software  and  operational  realism  topped  the  list  of  Air  Force  improvement  areas.  On  the 
contrary,  improvement  areas  for  AMC  reflected  a  different  focus.  For  mobility  programs,  spikes  centered  around 
resources  and  schedule.  The  authors  determined  that  limited  test  resources,  in  the  form  of  aircraft  sourcing, 
manning,  and  early  program  involvement  were  constraining  development  efforts,  resulting  in  unrealistic  schedules, 
unstable  software  systems,  and  immature  systems  integration.  A  study  conducted  by  RAND  further  concluded  that 
shortages  in  operational  test  manpower  currently  serve  as  a  barrier  to  realistic  and  early  testing. 13  The  correlation 
between  manning,  schedule  and  resources  are  complex.  However,  the  authors  concluded  this  T&E  shortage  poses  a 
significant  impact  on  AMC’s  ability  to  achieve  critical  knowledge  at  key  program  milestones.  This  analysis 
provided  key  indicators  that  an  AMC  T&E  transformation  could  reduce  some  of  the  instability  encountered  in  major 
modern/recap  programs. 

In  the  process  of  analyzing  problem  areas  in  the  AMC  T&E  Enterprise,  the  authors  concluded  that  findings  could 
be  indicators  of  a  greater  Air  Force  T&E  Enterprise  problem.  In  a  recent  address,  The  Honorable  Sue  Payton,  Under 
Secretary  of  the  Air  Force  for  Acquisition  remarked,  “We  are  a  country  at  war,  and  we  have  to  find  the  money  to  do 
the  things  we  have  to  do,  like  Precision  Airdrop.  We  do  not  have  a  seamless  acquisition  system.  It  is  a  build-up  of 
stovepipes.  But  we  are  working  very  hard  to  build  a  seamless  system.”14  With  these  remarks  in  mind,  the  authors 
sought  to  create  a  new  way  of  thinking  to  build  a  stronger,  more  seamless  connection  between  the  T&E  Enterprise 
and  the  joint  fight.  Central  to  this  new  way  of  thinking  is  changing  the  perception  of  the  T&E  Enterprise  as  a  cost 
increasing  endeavor. 

E.  Building  a  New  Way  of  Thinking  using  Effects-Based  Mobility  and  AFS021 

The  authors  used  two  methods  to  develop  a  way  to  transform  AMC  T&E.  The  first  method  involves  a  new 
cutting-edge  doctrine  in  AMC  called  Effects-Based  Mobility,  or  EBM.  EBM  leverages  mobility  systems  in  order  to 
generate  greater  effects  for  the  joint  fight.  It  pushes  leaders  to  select  “CRITICAL  FACTORS”  that  connect 
organizational  output  to  actual  battlefield  effects.15  While  the  systems-thinking  concepts  in  EBM  are  very  close  to 
Effects-Based  Operations  common  in  the  Air  Force  strike  community,  EBM  adds  tools  specific  to  Mobility  Air 
Force  (MAF)  needs  in  order  to  maximize  joint  effects.  The  authors  married  this  concept  to  AFS021,  which 
combined  a  review  of  best  business  practices  in  T&E  as  well  as  specific  AFS021  optimization  tools  in  order  to 
eliminate  waste,  “right  size,”  and  organize  the  future  AMC  T&E  force.  When  used  together,  EBM  provided  focus 
on  critical  factors  connecting  T&E  to  the  joint  fight,  while  AFS021  provided  the  tool  bag  to  quantify  improvement 
areas. 

F.  EBM  Connects  T&E  to  the  Joint  Fight 

Effects-Based  Mobility  compels  leaders  to  establish  a  systems  approach  to  generating  effects  for  Joint  Forces. 

For  AMC,  it  starts  with  the  commander’s  intent,  provided  by  General  Duncan  McNabb,  the  former  AMC 


Hardware  Software  Op  Realism  Schedule  Tng/Manuals  Resources  Comm 
Figure  4.  FY06  DOT&E  Improvement  Areas  for  27  Air  Force  Programs. 12 
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Commander. 16  In  AMC,  Velocity  and  Precision  are  critical  enablers  that  generate  effects  for  the  joint  fight  at  the 
Strategic,  Operational,  and  Tactical  level  of  war  (see  Figure  5). 

As  you  can  see  from  Figure  5,  the  top-down  analysis  started  with  a  review  of  the  effects  AMC  generates  for  the 
joint  fight.  As  EBM  dictates,  organizations  must  first  determine  critical  factors  that  generate  effects.  In  AMC, 
VELOCITY  and  PRECISION  enable  mobility  forces  to  generate  the  effects  shown  on  the  right  side  of  Figure  5. 
The  authors  determined  the  AMC  T&E  Enterprise  enables  VELOCITY  and  PRECISION  by  verifying  the  mission 
effectiveness  of  key  weapon  systems  used  by  mobility  forces.  Therefore,  Mission  Impact  was  identified  as  a 
critical  factor.  At  the  same  time,  AMC  T&E  also  has  a  profound  impact  on  the  cost  of  acquisition  programs. 
Finding  and  correcting  defects  early  in  the  acquisition  life  cycle  of  a  system  can  significantly  reduce  the  cost  of  a 
program.  Therefore,  Program  Cost  was  identified  as  another  critical  factor.  Identifying  Mission  Impact  and 
Program  Cost  as  critical  factors  focused  the  study  to  those  areas  that  enable  greater  effects  for  the  joint  fight. 
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Figure  5.  MAF  Effects  and  Commander’s  Intent  Formed  the  Basis  for  Analysis. 


G.  Using  AFS021  to  “Right  Size”  the  AMC  T&E  Enterprise 

So  far  in  the  study,  several  sub-optimum  areas  were  uncovered  in  the  T&E  Enterprise  (primarily  manpower  and 
resources,  as  shown  in  Figure  4).  EBM  connected  these  sub-optimum  areas  to  the  commander’s  intent,  and 
AFS021  pinpointed  manpower  and  resources  as  potential  sore  spots;  so  what  now?  Figure  3  provides  clues. 
Shifting  AMC  manpower  forward  in  the  acquisition  process  could  provide  significant  cost  gains  in  major 
modem/recap  programs.  Will  simply  adding  manpower  across  the  board  maximize  the  output  of  the  organization? 
How  much  manpower  should  be  added?  Will  returns  outweigh  the  investment?  By  how  much?  Quantifying  these 
numbers  with  a  credible  business  case  could  unlock  the  key  to  establishing  a  greater  bond  between  program 
managers  and  the  T&E  community.  In  other  words,  how  do  we  know  the  returns  are  worth  the  investment  in  T&E? 

In  order  to  answer  these  complex  questions,  the  authors  blended  several  methods  rooted  primarily  in  the 
operation’s  management  field.  The  authors  first  modeled  the  AMC  T&E  Enterprise  using  a  linear  programming 
technique  called  the  Analytical  Hierarchy  Method.17  The  model  allowed  leaders  to  quantify  decision-making 
priorities  to  maximize  mission  impact  and  reduce  cost.  By  modeling  the  AMC  T&E  Enterprise  from  a  manpower 
perspective  and  matching  that  model  to  the  critical  factors  of  Mission  Impact  and  Program  Cost,  an  optimized 
organizational  build  could  be  established.  Most  importantly,  the  results  could  be  quantified  in  order  to  provide 
senior  leaders  with  a  baseline  for  comparison,  to  include  cost  estimation.  Equation  1 ,  shown  below,  describes  the 
model,  the  total  output  in  annual  hours  of  the  AMC  T&E  Enterprise. 


6 

American  Institute  of  Aeronautics  and  Astronautics 


n 


M  =  X  (Ti  +  T2+  ...  +  T„)  1  +  (Ti  +  T2+  ...  +  Tn)2+...+  (T[  +  T2+  ...  +  Tn)K 

K=1 


(Equation  1) 


Where: 

M  =  The  total  time  per  year  devoted  to  test  program  planning,  execution  and  reporting 
T  =  Hours  devoted  by  a  member  of  AMC/TE  toward  an  AMC  Test  Program  in  1  year 
K  =  Number  of  organizations  in  the  AMC/TE  organization 

N  =  Number  of  members  in  each  AMC/TE  squadron,  detachment,  or  operating  location 

M  is  the  total  output  of  the  AMC  T&E  Enterprise  in  hours/year  devoted  to  a  list  of  44  current  projects  in  AMC. 
The  total  represents  only  hours  dedicated  to  project  planning,  execution,  and  reporting.  It  did  not  count 
administrative  overhead,  support  hours,  or  other  activities  associated  with  T&E  organizations.  Equation  2  is  the 
normally  distributed  density  function  of  the  product  of  Mission  Impact  and  Program  Cost,  the  two  critical  factors 
identified  as  critical  factors. 


F(x)  =  1  (xjiik) 
jli  a 


(Equation  2) 


Where: 

F  =  The  normally  distributed  hierarchy,  x,  between  mission  impact  and  program  cost 
jll  =  The  mean  of  the  probability  density  function 
a  =  Standard  Deviation  of  the  probability  density  function 

x  =  The  product  of  critical  factors,  based  on  an  hierarchy  index,  of  1-10  (see  Figure  6) 

The  product  of  the  critical  factors,  x,  was  achieved  by  reviewing  AMC  programs  with  respect  to  the  critical 
factors  and  conducting  a  weighted  comparison  of  program  priorities.  As  shown  in  Figure  6  (left  side),  some 
programs  like  Aeromedical  Evacuation,  rate  high  on  mission  impact  but  are  relatively  low  in  cost.  Others,  like  KC- 
X,  rate  high  in  both  mission  impact  and  program  cost.  When  normally  distributed,  the  majority  of  AMC  T&E 
workload  consists  of  mid-range  values,  as  shown  in  Figure  6  (right  side). 

(M  x  F)current  org  —  (M  x  F)new  org 

Output  improvement  (%)  =  _  (Equation  3) 

(M  x  F)current 

org 

Then  the  authors  calculated  the  product  of  M  and  F,  which  produced  a  non-dimensional  representation  of  the 
total  output.  Using  Equation  3,  the  authors  compared  the  output  of  various  organizational  builds  by  percentage. 
Maximizing  percentage  output  completed  the  model  and  provided  decision-making  options  based  on  critical  factors. 
Since  the  change  in  percentage  was  based  on  man-hours  dedicated  to  projects,  percentage  increases  were  converted 
back  to  man-hours  in  order  to  quantify  cost  savings. 

The  authors  ran  the  model  with  a  variety  of  organizational  builds  and  compared  output  to  the  current  state. 
Overall,  positioning  AMC  T&E  personnel  toward  earlier  involvement  optimized  this  complex  system.  When 
compared  to  the  original  plan,  using  the  Analytical  Hierarchy  Method  allowed  AMC  to  increase  output  by  17  %. 

The  fidelity  of  the  model  lies  with  the  authors’  understanding  of  the  level  of  effort  provided  to  each  program.  In 
other  words,  the  numbers  provided  did  not  include  a  sum  total  of  hours  worked  by  each  individual  in  a  given  year. 
Such  a  value  would  not  have  the  sensitivity  to  influence  Mission  Impact  and  Program  Cost.  Rather,  the  authors  only 
included  hours  of  direct  support  provided  by  AMC  testers.  Administrative  positions  and  support  activities  reduced 
the  sensitivity  of  the  model,  so  they  were  not  included  in  the  simulation. 

The  authors  also  developed  greater  efficiencies  in  the  resourcing  of  test  aircraft.  Since  mobility  aircraft  are 
considered  high-demand  assets,  minimizing  aircraft  absence  from  the  warfighter  is  key  to  increasing  the  critical 
factor  Mission  Impact.  Improved  aircraft  sourcing  techniques,  to  include  greater  participation  in  integrated  DT/OT 
efforts,  reduced  AMC  flight  test  hours  by  30%.  Added  to  this  was  an  effort  to  integrate  knowledge  management 
tools  in  order  to  forecast  future  workload  and  combine  test  efforts  when  possible. 

An  important  aspect  of  this  transformation  is  that  it  was  not  conducted  in  a  vacuum.  Throughout  the  study,  the 
authors  connected  with  Headquarters  Air  Force  AFS021  core  process  teams  and  utilized  knowledge  management  to 
integrate  with  AFS021  T&E  efforts  in  AFMC  and  AFOTEC.  They  conducted  several  meetings  with  experts  in  the 
aerospace  industry  and  government  organizations  like  NASA,  the  FAA,  and  RAND.  AMC  also  conducted  a  review 
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of  industry  best  practices,  including  a  review  of  T&E  successes  on  the  Boeing  777  program,  as  well  as  Intel™ 
software  improvement  programs. 18  These  efforts  enabled  the  authors  to  conclude  that  a  small  investment  in  AMC 
T&E  Transformation  could  yield  large  program  savings. 
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Figure  6.  Weighted  analytical  hierarchy  of  T&E  critical  factors  aimed  at  maximizing  joint  effects 

The  AMC  Transformation  plan  will  result  in  a  lean  and  efficient  organizational  design  that  positions  AMC 
involvement  earlier  in  the  acquisition  life  cycle  of  a  program.  Program  office  liaisons  were  developed  for  early 
involvement  at  product  and  logistics  centers  for  KC-X,  KC-135  and  C-5  modernization  programs.  More  positions 
were  moved  to  Edwards  AFB  to  place  greater  emphasis  on  integrated  DT/OT  programs  and  maximize  the  use  of 
scarce  resources.  Additionally  AMC  teamed  with  the  Air  Reserve  Component  in  order  to  leverage  expertise  in 
tactics,  defensive  and  communication  systems.  The  manpower  plan  incorporated  a  phased  approach  in  order  to 
retain  the  flexibility  to  move  personnel  as  workload  levels  change. 


Natick 
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Figure  7.  AMC  Transformation  (yellow  boxes  denote  new  locations  and  primary  function)19 

The  AMC  T&E  Transformation  represented  more  than  just  adding  personnel.  It  was  a  broad  transformation  effort 
designed  to  build  a  lean,  flexible  organization  capable  of  reducing  cost  for  the  modem/recap  programs. 
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H.  Results:  A  Lean  T&E  organization  Reduces  Modern/Recap  Cost 

The  authors  originally  determined  that  64  positions  (at  a  total  cost  of  $87  million  from  FY08  to  FY13)  were 
required  to  meet  AMC  T&E  workload  requirements  from  FY08  to  FY13.  However,  using  AFS021  achieved  a  17% 
increase  in  efficiency,  and  the  same  amount  of  workload  could  be  accomplished  using  only  44  positions  ($70 
million  investment).  Together,  the  authors  determined  that  the  transformation  could  meet  AMC  modem/recap 
requirements  by  adding  44  positions  and  5  new  organizations  throughout  the  CONUS,  as  shown  in  Figure  7.  Use 
of  AFS021  saved  20  positions  and  reduced  cost  by  approximately  $17  million.  Furthermore,  the  improved  flight 
sourcing  approach  (to  include  added  integrated  DT/OT)  netted  an  additional  $  1 5  million  savings  in  AMC  flight 
hours.  The  savings  totaled  $32  million.  However,  even  larger  savings  in  this  transformation  lie  with  the  nature  of 
T&E  and  its  ability  to  identify  system  defects  early  in  the  acquisition  process. 

In  order  to  explain  how  these  large  savings  occur,  reference  Figure  8.  As  large  acquisition  programs  progress 
through  early  acquisition  milestones,  testing  uncovers  large  lists  of  deficiencies,  depicted  by  the  red  bars  in  Figure  8. 
At  the  same  time,  capabilities  are  released  incrementally  up  to  and  beyond  initial  operational  capability  of  a  weapon 
system,  shown  by  the  blue  bars.  It  is  important  to  note  the  contractor  resolves  the  majority  of  the  deficiencies  early 
in  a  program,  while  deficiencies  found  after  fielding  result  in  costly  retrofits  and  larger  price  tags  for  block 
upgrades. 

The  lists  of  deficiencies  for  large  programs  number  in  the  hundreds.  However,  large  savings  are  embedded  in 
these  lists.  Approximately  5  to  10  percent  of  the  deficiencies  are  what  the  authors  termed  “Gems,”  or  high-dollar 
deficiencies  uncovered  by  testers.  When  Gems  are  found  early,  fixes  are  cheaper,  and  it’s  easier  to  leverage 
contractor  solutions.  For  example,  on  a  C-5  engine  upgrade,  an  AMC  tester  uncovered  the  potential  for  low 
reliability  and  leveraged  the  contractor  for  a  $250M  redesign  cost.  This  also  resulted  in  an  overall  system  reliability 
improvement  of  5%  on  the  C-5. 
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Figure  8.  Testers  reduce  deficiencies,  increase  capability,  and  reduce  cost 

As  the  system  moves  forward  in  development,  deficiencies  either  turn  into  costly  retrofits  or  add-ons  to  block 
upgrade  price  tags.  This  can  ultimately  result  in  capability  gaps  for  the  warfighter.  Capability  gaps  are  created 
when  deficiencies  progress  beyond  fielding.  In  2005  the  C-5  AMP  program  faced  a  complete  stop  of  Operational 
Testing  due  to  a  variety  of  system  deficiencies,  but  primarily  due  to  lagging  technical  order  development.  This 
resulted  in  a  lack  of  mission  ready  C-5  AMP  aircraft  delivered  to  AMC.  Problems  were  attributed  to  overtasked 
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T&E  manpower.  In  this  example  cascading  effects  of  an  excessively  lean  test  force  resulted  in  program  delays  and 
capability  gaps  for  the  warfighter. 

It  is  important  to  note  the  AMC  T&E  transformation  strategy  proposes  a  structure  that  finds  more  Gems,  but  on 
the  left  side  of  Figure  8.  That’s  where  program  managers  can  “buy  down”  program  risks  that  generate  cost  savings 
and  avoid  program  delays.  The  random  nature  of  these  Gems  make  them  seem  elusive,  but  the  analysis  shows  that 
casting  out  a  larger  T&E  “net”  in  the  right  spots  yields  large  savings.  This  prompts  an  important  question:  How 
many  gems  are  projected  with  44  additional  testers?  With  a  high  degree  of  confidence,  if  just  3  gems  are  found  on 
KC-X  and  the  Joint  Cargo  Aircraft  each,  and  2  Gems  on  the  remaining  programs  listed  in  Figure  8,  an  estimate  of 
the  magnitude  of  impact  testers  have  on  programs  totals  $300  million.  Realistically,  many  in  the  T&E  community 
expect  more  Gems  to  be  found,  but  the  numbers  shown  reflect  extremely  high  confidence.  The  analysis  provides  a 
compelling  business  case  and  shows  that  an  efficient  T&E  force  significantly  reduces  cost  on  modem/recap 
programs. 

I.  Transformation  Results 

Surely  this  study  was  not  accomplished  just  to  determine  if  early  user  involvement,  integrated  DT/OT,  and  use  of 
modeling  and  simulation  is  a  value-added  endeavor.  Pundits  professing  these  concepts  have  been  doing  so  since  the 
Packard  Commission  in  1986  (the  commission  established  the  framework  for  current  acquisition  management 
structure).  From  Lightning  Bolt  Initiatives  to  recent  Acquisition  Reform  laws,  policy  makers  have  unveiled  “new” 
concepts  in  T&E  policy  nearly  every  five  years  since  1986.  Yet  DOD  acquisition  programs  continue  to  suffer  from 
significant  cost  overruns  and  are  the  subject  of  repeated  congressional  inquiry. 

The  AMC  Test  Transformation  drove  directly  to  the  heart  of  program  instability  to  find  specific  T&E  links,  and 
subsequently  uncovered  techniques  testers  can  use  to  transform  their  organizations.  In  this  study,  actions  hinged  on 
maximizing  mobility  effects.  The  authors  did  this  by  building  a  stronger  connection  between  the  T&E  Enterprise 
and  joint  effects.  In  AMC,  the  authors  used  EBM  to  connect  the  T&E  Enterprise  to  command  priorities.  They  also 
broke  out  AFS021  tools  and  used  a  variety  of  techniques  to  pinpoint  organizational  weaknesses.  In  the  course  of 
this  study,  the  authors  also  uncovered  findings  systemic  to  the  Air  Force  T&E  Enterprise.  The  following 
summarizes  the  primary  findings: 

1.  T&E  is  linked  to  the  stability  of  a  weapon  system  design,  and  ultimately  provides  “knowledge  early”  for  program 
managers  to  make  key  milestone  decisions. 

2.  A  lean,  efficient  T&E  force  sized  and  positioned  properly  for  projected  workload  reduces  program  costs  primarily 
through  the  discovery  of  “GEMS,”  or  high  dollar  defects;  ultimately  it  serves  as  a  direct  investment  into 
modern/recap  programs. 

3.  In-depth  analysis  using  industry-approved  operations  management  techniques  builds  confidence  in  the  ability  of 
T&E  to  perform  its  intended  function. 

4.  Testers  must  use  a  broad  approach  to  quantify  effects  for  decision-makers  and  effectively  articulate  the 
magnitude  of  impact  T&E  can  have  on  modern/recap  programs. 

5.  Ultimately,  connecting  T&E  to  the  joint  fight  is  a  worthwhile  endeavor,  because  it  provides  a  basis  for  showing 
how  T&E  resources  bring  back  “returns”  for  the  warfighter  and  the  taxpayer. 

This  brings  us  to  the  original  question  posed  in  the  beginning  of  this  paper.  Why  should  PMs  invest  in  T&E 
transformation?  It’s  simple.  A  well-placed  investment  in  T&E  can  prevent  a  $34  billion  bill  from  becoming  a 
$50  billion  over-budget  and  delayed  list  of  programs.  More  importantly,  upfront  investment  in  T&E 
organizations  reduces  program  cost,  stabilizes  designs,  and  helps  program  managers  “buy  down”  the  risk  of  a 
weapon  system  program.  However,  the  T&E  community  must  demonstrate  good  stewardship  of  this  investment 
using  valid  business  tools.  Another  key  message  is  in  the  T&E  way  of  thinking  and  how  it  is  articulated  to  program 
managers:  All  in  all,  SYSTEM  DEFECTS  ARE  GEMS  TO  BE  MINED,  and  form  the  core  antibodies  against 
cascading  negative  program  effects. 

Finally,  a  fundamental  conclusion  from  this  study  is  that  modernization  and  recapitalization  programs  benefit 
when  the  T&E  community  is  “right  sized”  for  efficiency.  This  way  of  thinking  forms  the  core  basis  for  building  the 
trust  of  warfighting  commands  and  program  managers.  This  effort  must  occur  on  an  annual  basis.  It  must  utilize  a 
broad  range  of  tools,  with  the  right  level  of  analysis. 

J.  A  Look  to  the  Future:  A  Truly  Seamless  Enterprise  Starts  with  Doctrinal  Change 

The  next  cutting-edge  step  in  seamless  T&E  is  the  replacement  of  Developmental  and  Operational  Test  with  a 
single  integrated  T&E  doctrine.  The  DT/OT  concept  is  an  antiquated,  Cold  War  model,  built  for  large 
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bureaucracies.  The  reality  of  spiral  development  and  the  speed  of  today’s  technological  advancements  may 
invalidate  the  need  for  a  separate  DT  and  OT  focus.  The  changing  global  landscape  and  the  Global  War  on  Terror 
dictates  that  a  responsive,  flexible  T&E  force  focused  on  joint  effects  could  maximize  efficiency  for  the  warfighter 
and  taxpayer. 

There  are  many  models  available  for  the  Air  Force  to  review  in  order  to  research  a  more  seamless  doctrine.  One 
potential  model  for  this  responsive  structure  is  the  Air  Force  weapons  testing  infrastructure.  Furthermore,  other 
countries  (for  example,  India’s  Test  Establishment  or  the  Israeli  Defense  Force)  also  work  in  resource-constrained 
environments  and  follow  efficient  T&E  models  dedicated  to  effects.  Industry,  NASA,  and  the  FAA  already 
subscribe  to  a  single  T&E  doctrine,  driven  partially  by  funding  constraints  and  the  need  to  build  greater  profit 
margins.  Studying  these  organizations  could  be  the  key  to  establishing  a  seamless  enterprise  in  light  of  current 
funding  constraints. 

Continually  catering  to  the  seam  between  mission  effectiveness  and  spec  compliance  (referring  to  Operational 
and  Developmental  testing)  adds  drag  to  T&E  programs,  and  necessitates  the  need  for  two  large  stove-piped 
organizations.  The  speed  of  acquisition  demanded  by  the  warfighter  in  the  Global  War  on  Terror,  and  the 
emergence  of  new  threats  in  todays  global  landscape  are  already  shaping  this  environment.  Regardless  of  the 
solution,  the  time  is  now  to  build  new  ways  of  thinking,  and  remove  sources  of  doctrinal  dogma  created  by  the 
constraints  of  previous  decades.  The  final  question  we  have  to  ask  ourselves  is:  are  we  as  a  service  ready  to  tum-in 
our  empires,  focus  on  the  point-of-effect  in  the  joint  fight,  and  build  a  truly  seamless  integrated  doctrine? 

All  of  us  need  to  focus  on  maximizing  effect  for  joint  forces .  This  is  not  about 
airplanes  or  ships.  It  is  about  responding  at  the  point  of  effect  for  theater 
commanders. 


General  Norton  Schwartz 
Commander,  USTRANSCOM 
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